Remove properties and cross_validation FilePaths from metadata#2869
Remove properties and cross_validation FilePaths from metadata#2869allisterakun merged 18 commits intodevfrom
properties and cross_validation FilePaths from metadata#2869Conversation
|
Current Coverage: % Mypy errors on remove_properties_fp_from_metadata branch: 1213 |
|
🚨 Please update the changelog. This PR cannot be merged until |
|
Current Coverage: 99% Mypy errors on remove_properties_fp_from_metadata branch: 1213 |
|
🚨 Please update the changelog. This PR cannot be merged until |
ew3361zh
left a comment
There was a problem hiding this comment.
LGTM, nice work Allister.
| "feed_storage_configurations", | ||
| "feed_storage_instances", | ||
| } | ||
| PRROPERTIES_FILE_PATHS: dict[str, Path] = { |
There was a problem hiding this comment.
Do you think docstrings would be helpful here? I know we're not doing it anywhere else in this file but we do it in manure files for file-level constants. Just a thought.
Co-authored-by: Niko <70217952+ew3361zh@users.noreply.github.com>
|
Current Coverage: 99% Mypy errors on remove_properties_fp_from_metadata branch: 1214 |
|
🚨 Please update the changelog. This PR cannot be merged until |
|
Current Coverage: 99% Mypy errors on remove_properties_fp_from_metadata branch: 1214 |
|
🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended. |
|
Current Coverage: 99% Mypy errors on remove_properties_fp_from_metadata branch: 1215 |
|
🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended. |
Removes
propertiesandcross-validationfile path definitions from metadata JSON files. Properties file paths are now defined as a module-level constant ininput_manager.py, and cross-validation file paths are moved to the task configuration JSON files.Context
Issue(s) closed by this pull request: closes #2859.
What
"properties"file blob entry from all metadata JSON files (freestall_e2e_metadata.json,no_animal_e2e_metadata.json,open_lot_e2e_metadata.json,example_freestall_dairy_metadata.json,example_no_animal_metadata.json,example_open_lot_metadata.json,available_simulations_metadata.json,end_to_end_testing_tm_metadata.json,herd_init_metadata.json,update_end_to_end_testing_tm_metadata.json,task_manager_metadata.json)."properties"fromREQUIRED_FILE_BLOBSininput_manager.py.PRROPERTIES_FILE_PATHSininput_manager.pythat maps property file names to theirPathobjects (default.json,tasks_properties.json,commodity_properties.json)._load_propertiesinInputManagerto iterate over the hardcodedPRROPERTIES_FILE_PATHSinstead of reading paths from the metadata."cross-validation"key from all metadata JSON files."cross_validation_file_paths"as a new field in task configuration JSON files (available_simulation_tasks.json,end_to_end_testing_task.json,example_freestall_task.json,example_no_animal_task.json,example_open_lot_task.json,single_run.json).start_data_processingto accept across_validation_file_paths: list[str] | Noneparameter and pass it to_cross_validate_data._cross_validate_datato receivecross_validation_file_pathsas a parameter instead of reading it fromself.__metadata.TaskManager.startandhandle_input_data_auditto passcross_validation_file_pathsthrough tostart_data_processing."default"tooptional_array_property_keysinDataValidator._metadata_array_validator.input/metadata/properties/tasks_properties.jsonwith thecross_validation_file_pathsproperty, where it is a nullable array of strings with default beingnull.test_input_manager.pyandtest_task_manager.py.Why
Previously,
propertiesandcross-validationfile paths were embedded inside each metadata JSON file. This led to issue #2859. Properties file paths are the same across all simulations, so hardcoding them as a constant simplifies metadata files and eliminates redundancy. Moving cross-validation file paths to the task configuration files gives each task explicit control over which cross-validation rules apply, which is a more appropriate location since different tasks may require different validation rules.How
For properties, the
_load_propertiesmethod now reads from thePRROPERTIES_FILE_PATHSconstant dictionary instead of extracting paths fromself.__metadata["files"]["properties"]. The"properties"entry is no longer needed in metadata files and has been removed, along with thedel self.__metadata["files"]["properties"]cleanup step.For cross-validation, the
"cross-validation"key is removed from metadata files entirely. Instead, each task JSON file now includes a"cross_validation_file_paths"field (a list of file path strings, or an empty list). This value flows throughTaskManager.handle_input_data_audit→args["cross_validation_file_paths"]→InputManager.start_data_processing→_cross_validate_data→_load_cross_validation.Test plan
test_input_manager.pyto reflect the newcross_validation_file_pathsparameter instart_data_processingand the removal of properties from metadata.test_task_manager.pyto passcross_validation_file_pathsin the appropriate call assertions.Input Changes
input/metadata/properties/tasks_properties.json."cross_validation_file_paths"field to all task JSON files (available_simulation_tasks.json,end_to_end_testing_task.json,example_freestall_task.json,example_no_animal_task.json,example_open_lot_task.json,single_run.json)."properties"file blob entry from all metadata JSON files."cross-validation"key from all metadata JSON files.Output Changes
Filter